Interactive projector system and method

ABSTRACT

An interactive image projecting device that projects an image onto a display surface and controls the projected image based upon movement of the image projecting device or the detection of another image. The image projecting device includes a projector that generates an image from a control unit. The device includes a position indicating emitter that generates a position indicator and an image sensor that is operable to detect the position indicator from either the device including the image sensor or a position indicator from another device. Based upon the sensed position indicator from another device, the control unit operates to modify the image projected from the device such that the image from a first device can interact with an image from a second device. The first and second devices each have a wireless transceiver such that the devices can communicate with each other during operation.

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation application of prior U.S. patentapplication Ser. No. 12/980,966, filed on Dec. 29, 2010, now issued asU.S. Pat. No. ______ entitled INTERACTIVE PROJECTOR SYSTEM AND METHOD,which is a continuation application of prior U.S. patent applicationSer. No. 11/867,913, filed on Oct. 5, 2007, now issued as U.S. Pat. No.7,874,681, entitled INTERACTIVE PROJECTOR SYSTEM AND METHOD.

FIELD OF THE INVENTION

The present invention generally relates to interactive videoentertainment devices. More specifically, the present invention relatesto interactive image projecting devices that control the image beingprojected based upon the orientation and movement of the projectingapparatus and based upon images projected by another image projectingapparatus.

BACKGROUND OF THE INVENTION

Presently, there have been many different types of interactive videoentertainment devices that allow a player/user to move a device orcontroller that results in movement of the game being played. One typeof highly popular video game console is the Wii game machine andcontroller manufactured by Nintendo, Inc. of Japan. This game systemenables a user to interact with a video game by waving or swinging awireless controller through the air. However, this type of game systemrequires a game machine, display monitor and controller to allow theplayer to interact with the gaming environment. Other types of hand heldelectronic entertainment devices are readily available that include verysmall display screens. These types of devices, although allowing theplayer to have increased mobility when playing the game, do not allowthe participant to interact with the video game either by moving acontroller or the device. Instead, the user simply depresses buttons orcontrol elements on the hand held gaming device to control operation ofthe gaming environment.

Presently, manufacturers and product developers are working on videoprojectors that are very small in size and can be embedded into otherdevices, such as video cameras or cell phones. The current focus ofthese projection systems is to provide a very small projector for use inprojecting video images, rather than on utilizing the projectors in agaming or entertainment device.

Therefore, an opportunity exists for the use of small and compact videoprojectors in a hand held device for use in playing games and otherinteractive opportunities. Further, an opportunity exists for the use ofsmall video projectors in combination with image sensors such that ahand held entertainment device can interact with other devices toprovide a unique interactive gaming experience.

SUMMARY OF THE INVENTION

The present invention generally relates to an interactive imageprojecting device in which the device generates an output image. Acontrol unit within the interactive image projecting device modifies theprojected image based upon either movement of the projecting device orthe detection of an image from another device. In one embodiment, theimage projecting device includes a microprocessor-based control unitthat is operatively associated with a projector for projecting an imagefrom the device. Preferably, the projector is a laser-based projector,although other types of projectors are contemplated.

In addition to the projector, the image projecting device also includesan image sensor that is operable to detect images, objects and movementin front of the image projecting device. As an example, the image sensormay be a CMOS camera that is able to detect at least infrared light in aregion in front of the image projecting device. The image sensor iscoupled to the control unit such that the control unit can respond toimages sensed by the image sensor.

In addition to the image sensor, the image projecting device preferablyincludes a position indicating emitter, such as an infrared LED.Preferably, the position indicating emitter emits an infrared beam oflight to define a position indicator on a display surface. The positionindicator may be of various shapes and patterns, but it is preferredthat an anisotropic shape be used such that rotation of the image can besensed.

The image projecting device can also include a spatial position sensorthat is mounted within the housing of the image projecting device and isoperable to generate a movement signal received by the control unit thatis based upon the movement of the housing. Preferably, the spatialposition sensor is an MEMS accelerometer, although other devices arecontemplated. Based upon the sensed movement signals from the spatialposition sensor, the control unit can modify the image from the deviceto simulate the movement of the image projecting device.

Preferably, each of the interactive image projecting devices includes awireless transceiver such that an image projecting device cancommunicate to other devices. The communication between a pair of imageprojecting devices allows the devices to interact with each other suchthat the images displayed by each device can react to the image beingdisplayed by the other device.

In one embodiment of the image projecting device, the device includes anilluminating emitter that can be activated to generate an illuminatinglight source. Preferably, the illuminating light is a source of infraredlight. When the illuminating emitter is activated, the illuminatingemitter creates an infrared “shadow” behind objects positioned in frontof the image projecting device. Based upon the presence or absence of ashadow, the control unit of the image projecting device can completedesired actions. As an example, the elimination of a shadow behind aplayer's hand indicates that the player has contacted a display surface.This contact hand gesture can be interpreted by the control unit tocause the control unit to modify its displayed image.

In addition to being used as a stand-alone unit, each of the interactiveimage projecting devices can be utilized with another device. In suchuse, the first device generates a first image and a position indicatorthat are displayed upon a display surface. At the same time, a seconddevice is displaying a second image and a second position indicator.During operation, the image sensor of the first device attempts tolocate the position indicator from the second device. When the imagesensor detects the position indicator from the second device, the imagesensor relays this information to the control unit. Upon sensing anotherdevice, the control unit of the first device modifies the image beingprojected from the first device such that the first image interacts withthe second image. At the same time, the second device can sense theposition indicator of the first device and adjust its image accordingly.In this manner, a pair of first and second devices can adjust theirprojected images such that the images appear to interact with eachother.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings illustrate the best mode presently contemplated of carryingout the invention. In the drawings:

FIG. 1 is a perspective view of a first embodiment of the interactiveimage projecting device of the present invention;

FIG. 2 is a top, internal view of the image projecting device of FIG. 1;

FIG. 3 is a side, internal view of the image projecting device;

FIG. 4 a is an illustration of a first and a second image projected by afirst and second image projecting devices;

FIG. 4 b illustrates the infrared camera view of the second imageprojecting device and the position indicator from the first imageprojecting device;

FIG. 4 c shows the infrared camera view from the first image projectingdevice and the position indicator from the second image projectingdevice;

FIG. 5 a illustrates the interaction between a projected image and areal world person;

FIG. 5 b shows the infrared camera view of the image projecting deviceand the proximity detection of a human figure;

FIG. 5 c shows the modified image based upon the detection of the humanfigure;

FIG. 6 a shows the top view in which a user is about to make a contacthand gesture to interact with the image from the image projectingdevice;

FIG. 6 b shows a rear view of the user about to make a contact handgesture;

FIG. 6 c shows the image sensor view of a user about to make a contacthand gesture;

FIG. 6 d shows the image sensor view when the illuminating emitter isactivated to create a shadow;

FIG. 6 e illustrates a user making contact with a display using a handgesture;

FIG. 6 f is a rear view of the user making contact with a hand gesture;

FIG. 6 g is the image sensor view when the user has made a contact handgesture;

FIG. 6 h is the image sensor view when the illuminating emitter isactivated;

FIG. 7 a is a flow diagram of the control unit during the calibration ofthe image projecting device;

FIG. 7 b is a flow diagram of the control unit when operating to detectanother device' projected image;

FIG. 8 a shows a user making a non-contact hand gesture to interact withthe projected image from the image projecting device;

FIG. 8 b illustrates a projected image from one of the image projectingdevices;

FIG. 8 c illustrates the view of the image sensor showing a user thathas made a hand gesture;

FIG. 8 d shows the resulting projected image following the user handgesture;

FIG. 9 a is a high-level flow diagram of the control unit that enablesuser hand gesture observation, detection, analysis, and response;

FIG. 9 b illustrates the steps of the control unit that enable the imageprojecting device to detect hand and shadow regions during both contactand non-contact gestures;

FIG. 9 c shows the steps of the control unit that enables the imageprojecting device to respond to a “chopping” gesture;

FIG. 10 is a diagram of the display field for the image projector andthe viewing field for the image sensor;

FIG. 11 illustrates a person interacting with a projected image on awall by making a “chopping” gesture;

FIG. 12 illustrates the image projecting device in the left hand of auser and a supplementary gesture glove worn on the right hand;

FIG. 13 illustrates the image projecting device held in the user's lefthand and a supplementary gesture device attached to the user's righthand;

FIG. 14 shows a top view of one embodiment of the supplementary gesturedevice shown in FIG. 13;

FIG. 15 a shows the projected image from one of the image projectingdevices;

FIG. 15 b shows the image sensor view from one of the image projectiondevices and the infrared position indicator from another of the imageprojecting devices;

FIG. 15 c shows the clipping of the projected visible image from onedevice;

FIG. 15 d shows the resulting display of a single monolithic imageprojected from a pair of image projecting devices;

FIG. 16 illustrates the image projecting device wirelessly communicatingwith an external device, such as a game machine connected to theinternet;

FIG. 17 shows a second, alternate embodiment of the image projectingdevice of the present invention;

FIG. 18 shows a top, internal view of the second, alternate embodimentof FIG. 17;

FIG. 19 shows a side, internal view of the second, alternate embodiment;

FIG. 20 shows a pair of images projected from the image projectingdevices of FIG. 17;

FIGS. 21 a and 21 b illustrate the image sensing views and positionindicators from the pair of the image projecting devices;

FIG. 22 shows a diagrammatic side section view of a third, alternateembodiment;

FIG. 23 shows a diagrammatic side view of a fourth, alternateembodiment;

FIG. 24 is a perspective view of a fifth embodiment of the interactiveimage projecting device of the present invention; and

FIG. 25 is a side, internal view of the image projecting device shown inFIG. 24.

DETAILED DESCRIPTION OF THE INVENTION

Referring first to FIGS. 1-3, thereshown is a first embodiment of animage projecting device 10 constructed in accordance with the presentinvention. The image projecting device 10 is designed to be of a sizethat can be held in the hand of a user. In the embodiment shown in FIGS.1-3, the image projecting device 10 includes an outer housing 12preferably formed from an injection-molded plastic material, such aspolypropylene, polystyrene or polycarbonate. The outer housing 12 in theembodiment shown is roughly 120 mm long by 80 mm wide by 25 mm thick insize, although other sizes are clearly contemplated as being within thescope of the present disclosure.

The outer housing 12 defines a front end 14, a pair of sides 16, a topsurface 18 and a back end 20. In the embodiment shown in FIG. 2, theback end 20 includes a plastic, vinyl or cloth strap 22 that allows theuser to attach the device 10 to the wrist or clothing so that the devicedoes not accidentally become separated from the user during use of thedevice.

Referring now to FIG. 2, the image projecting device 10 includes acentral control unit 24 that controls the operation of the entiredevice. In the embodiment shown in FIG. 2, the control unit 24 is amicroprocessor that is capable of operating the components to bedescribed in detail below. The control unit 24 includes aself-contained, internal power supply, such as a series of batteries 26that are self-contained within the outer housing 12. The batteries 26provide operational power for the remaining components to be described.

Further shown in FIG. 2, a data storage unit 37 is coupled to thecontrol unit 24 enabling the reading and writing of data duringoperation. The storage unit includes dynamic memory along with a harddrive, disk player, or Flash memory cartridge, although other kinds ofdata storage are clearly contemplated as being within the scope of thepresent disclosure.

The image projecting device 10 includes a speaker 28 for playing soundsfrom a sound synthesizer integrated circuit 30. The sound synthesizer 30receives signals from the control unit 24 to control the sounds playedby the image projecting device 10. In the embodiment illustrated in FIG.2, the device includes a small, vibrator motor 32 that can be activatedby the control unit 24 to cause the entire device 10 to vibrate asdesired. In addition, the device 10 contains a microphone 49 that issensitive to ambient sound, such as a player clapping or a door knocksound. The microphone 49 is operatively coupled to the control unit 24,enabling a player to interact with the image projecting device 10 usingsound. An on/off switch 34 is positioned within the outer housing 12 andis accessible from the top surface 18. The on/off switch 34 is shown asbeing a push-button, although other types of switches are clearlycontemplated as being within the scope of the present disclosure.

The top surface 18 further includes a track ball 36 that is in operativecommunication with the control unit 24. The track ball 36 allows theplayer to enter movement information into the image projecting device 10such that the image projecting device can control movement of a displayimage based upon signals entered by a player through the track ball 36.

Referring now to FIG. 2, the image projecting device 10 includes aspatial position sensor 38 that is operatively connected to the controlunit 24. In the preferred embodiment of the invention, the spatialposition sensor 38 is a MEMS accelerometer that provides movementsignals to the control unit 24. Preferably, the MEMS accelerometerprovides information to the control unit regarding movement of theentire image projecting device 10 along x, y and z axes. Through theinformation received from the spatial position sensor 38, the controlunit 24 is able to determine relative movement of the entire imageprojecting device 10 by the player.

As can also be seen in FIG. 2, the image projecting device 10 includes awireless transceiver 40 that is operatively coupled to the control unit24. The wireless transceiver 40 allows for two-way communication betweenthe image projecting device 10 and another similar image projectingdevice 10. Additionally, the wireless transceiver 40 allows for wirelesscommunication between the image projecting device 10 and other wirelesscommunication devices, such as a personal computer or other device thatis able to communicate through wireless transmission. It is contemplatedthat the wireless transceiver 40 could communicate using variousdifferent wireless transmission techniques, such as Bluetooth, RF,ZigBee or any other similar wireless standard.

The image projecting device 10 further includes a projector 42 that isin operative communication with the control unit 24 and is operable toproject an image from the image projecting device 10. As illustrated inFIGS. 1 and 2, the projector 42 extends through the front end 14 suchthat the projector 42 is able to project an image forward of the frontend 14. In the embodiment of the invention shown in FIGS. 1-3, theprojector 42 is a laser-based projector that projects a visible colorimage through a transparent cover 44. The transparent cover 44 isretained within a circular housing 46. Unlike standard incandescent bulbprojecting systems, the laser-based projector 42 does not require anyfocusing lens since the projected image is always in focus, irrespectiveof the projector to wall distance. One desirable property of thelaser-based projector 42 of the first embodiment is the ability toclearly display an image on an irregular background of objects, such asa chair near a wall. Since the image from the laser-based projector isalways in focus, the image that falls on both the chair and the wallwill be in focus. Although a laser-based projector 42 is shown anddescribed in the first embodiment, it should be understood that othertypes of image projecting systems, such as those that incorporate anincandescent bulb and a focusing lens, are contemplated as being withinthe scope of the present invention.

In the embodiment shown in FIGS. 1-3, the laser-based projector 42 has alight divergent angle of approximately 20°-60° to define an imageprojection field, as shown by reference numeral 48 in FIG. 10. The imageprojection field 48 is shown contacting a display surface 50 such thatthe image has a width W, which is determined by the divergent angle ofthe projector and the distance of the image projecting device 10 fromthe display surface 60.

Referring back to FIG. 2, the image projecting device 10 furtherincludes a position indicating emitter 52 that is operable to generateand project a position indicator from the front end 14 of the imageprojecting device. In the embodiment shown in FIG. 2, the positionindicating emitter includes an infrared light emitter, such as infraredLED 54, that is positioned within a waveguide 56. In the embodimentshown, the waveguide 56 is an anisotropic-shaped focusing waveguide thatproduces a focused, infrared beam from the infrared LED, typically of1°-20° of divergence, which projects forward from the device 10 andilluminates an ambient surface. The infrared LED 54 produces infraredlight having an approximate wavelength of 880-940 nanometers, which iscommonly available in the current marketplace.

In the embodiment shown in FIG. 2, the anisotropic-shaped focusingwaveguide 56 is molded into the housing 12 and acts as a tube or guidefor light waves from the LED 54. One advantage of such a design approachis that the divergence angle of the light from the LED 54 can becontrolled independently of the light divergence angle of the infraredLED 54. Such a design ensures that each of the image projecting devices10 will produce the same light divergence angle, irrespective ofpotential slight variations in the light-sensitive hardware componentcharacteristics. In addition, the anisotropic-shaped focusing waveguide56 produces an anisotropic-shaped area of infrared light on the ambientsurface.

As shown in FIG. 4 b, the position indicating emitter emits light at thedivergence angle shown by dashed lines 58 to create the positionindicator 60. In the embodiment shown in FIG. 4 b, the positionindicator 60 has a “T” shape, although other anisotropic patterns oranisotropic-shaped position indicators 60 are contemplated as beingwithin the scope of the present disclosure. Further, other types ofemitters are also contemplated as being within the scope of theinvention, as are other wavelengths of light being emitted. For example,the laser projector 42 could contain an infrared laser emitter,providing a means to produce the position indicator.

Referring back again to FIG. 2, the image projecting device 10 furtherincludes an image sensor 62 that is operatively coupled to the controlunit 24. The image sensor 62 is operable to detect images that areprojected in front of the image projecting device 10. Preferably, theimage sensor 62 has a field of view that is wider than the projectionfield for the image projecting device 10. Referring now to FIG. 10, theimage viewing field 64 for the image sensor is shown as being greaterthan the image projecting field 48. In this manner, the image projectingdevice 10 can sense an area greater than the area over which the imageprojecting device 10 can project an image.

Referring back to FIG. 2, in the preferred embodiment of the invention,the image sensor 62 is a CMOS camera 66 positioned behind an opticallens 68 that is mounted within the front end 14 of the outer housing 12.The optical lens 68 is mounted within a camera housing 70 that can, inthe embodiment shown in FIG. 2, include a filter 72 designed to filterout undesirable light. In the embodiment shown in FIG. 2, the filter 72is an infrared bandpass filter that is transparent to infrared light andopaque to all other types of light, such as visible light. The opticallens 68 is capable of transferring both infrared light and visible lightand is preferably made of materials such as polypropylene, polyethyleneor glass. Further, other types of image sensors are also contemplated asbeing within the scope of the invention, such as a CCD sensor, thermalsensor, or photo diode array.

In the embodiment shown in FIG. 2, the CMOS camera 66 is operativelyconnected to the control unit 24 and contains a complementarymetal-oxide semiconductor (CMOS) sensor, typically having a globalsynchronous electronic shutter, along with an approximate view angle of40°-120°. Examples of CMOS cameras 66 are available from variousmanufacturers, such as Texas Instruments, Toshiba, and Fujitsu, althoughvarious other types of the CMOS camera are well known in the art.

Referring now to FIG. 1, the image projecting device 10 further includesa wide angle illuminating emitter 74 that is mounted just behind thefront end 14 of the outer housing 12. The illuminating emitter 74 ispositioned below the image sensor 62 and is operable as an illuminatingdevice for illuminating objects in front of the image projecting device.As an example, the illuminating emitter 74 can be a wide-angle infraredemitter having a divergence light angle of 40°-120° and having anapproximate wavelength of 880-940 nanometers. Such an infrared emitteris commonly available in the marketplace. For greater illumination, aplurality of emitters is also contemplated as being within the scope ofthe invention.

Referring back to FIG. 1, the image projecting device 10 furtherincludes a range locator sensor 51, mounted just behind the front end 14of the outer housing 12. The range locator sensor 51 is operativelyconnected to the control unit, providing a means to determine thedistance to the ambient surface on which the projected image resides.The range locator sensor 51 is preferably a narrow beam, infrared rangelocator having a sensitivity range of about 0.5 to 6 feet, such as theSharp GP2Y0A02YK IR sensor, although other types of range locators arecontemplated as being within the scope of the invention. As an example,the combination of the position indicating emitter 52 and the imagesensor 62 could be configured to determine the range to the projectionsurface based upon triangulation or “time-of-flight” of the light path.

Referring now to FIGS. 4 a-4 c, the operation of a first imageprojecting device 10 a and a second image projecting device 10 b will bedescribed. As illustrated in FIG. 4 a, the first device 10 a isprojecting a first image 76, such as a cat, while the second device 10 bis projecting a second image 78, such as a dog. Although first andsecond images 76, 78 are shown in FIG. 4 a, it should be understood thatthe devices 10 a, 10 b could be programmed to display any type of imagedesired.

As shown in FIG. 4 b, the first device 10 a also projects a positionindicator 60 a while the second device 10 b has an image viewing field64 b for the image sensor. As shown in FIG. 4 b, the image sensor forthe second device 10 b can detect the position indicator 60 a from thefirst device 10 a since the position indicator 60 a is within the imageviewing field 64 b.

At the same time, the image sensor of the first device 10 a has an imageviewing field 64 a that can detect the presence of the positionindicator 60 b from the second device 10 b. In this manner, the firstand second image projecting devices 10 a, 10 b can detect the presenceof the other device and determine the position of the position indicator60 of the other device relative to the image viewing field 64. Basedupon the detected location of the image from the first device 10 arelative to the second device 10 b, the control unit of the respectivedevices can modify the image produced by the device such that the images76, 78 appear to interact with each other.

Although the image projecting devices 10 shown in FIGS. 4 a-4 c aredescribed as being able to interact with each other, each of the imageprojecting devices 10 could be utilized separately. In a use withoutanother device, the user first depresses the on/off switch 34 to turnthe device on. Once the device is on, the user points the device towarda wall or other similar surface roughly 1-6 feet away to observe aprojected image of a character on the wall. Preferably, the projectedimage would typically range in size from 1-6 feet in diameter, dependingupon the distance from the device to the display surface.

Since the device 10 includes the x, y, z axis spatial position sensor38, the device 10 can determine the relative motion of the device in 3Dspace as the player moves the device in an arc, zigzag, or a straightmotion. The control unit 24 reads the output of the spatial positionsensor 38 and animates the projected image accordingly. For example, ifa player sweeps or moves the device along the wall in a left horizontaldirection, the projected image moves in an animated manner to the left.Likewise, if the player sweeps the projected beam of light along thewall in a right horizontal direction, the projected image moves in ananimated manner to the right. Further, as one moves the projected beamof light along the wall and onto the ceiling, the projected image turnsaround and appears to be walking away from the player. Finally, if theplayer sweeps the image down toward the floor, the projected image turnsaround and appears to be walking toward the user.

It is anticipated that the control unit 24 can accomplish variousmovements of the image based upon signals received from the spatialposition sensor 38. As an example, if the control unit determines thedevice is tilted with a quick, upward flick of the wrist, the projectedimage may be revised to cause the image to perform a certain action,such as having a dog sit up on its hind legs. If the device is tilteddownward with a quick, sudden movement of the wrist, the projected imagecould perform another action, such as causing a dog to lie down.

Additional interaction capabilities are obviously within the scope ofthe present disclosure and depend upon the type of character being shownin the image and the movement of the image projecting device. Further,since the device 10 includes other input devices, such as the track ball36 shown in FIG. 2, the control unit can be operated to cause the imageto perform certain functions depending upon the movement of the trackball 36 with the player's finger. The track ball 36 can be used to movea portion of the image, such as a cursor or other indicator, along theimage. Once the cursor or other indicator is in the desired position,the projecting device 10 can cause the image to perform another action,such as a dog picking up a bone.

Referring now to FIG. 7 a, before the image projecting devices 10 areused to interact with each other, and periodically while the devices areinteracting with each other, each of the image projecting devices goesthrough a self-calibration process shown by the flowchart of FIG. 7 a.As illustrated in FIG. 7 a, the control unit for the image projectingdevice first determines the distance to the projection surface using therange locator, as shown in step 80. Subsequently, the control unit thencomputes look-up table index I in step 81. If item I within the look-uptable already contains data, as shown in step 82, all correction Δvalues are set in step 83. If the device (at this projection distance)has not been calibrated, the control unit turns on the positionindicating emitter to generate the position indicator, as described instep 84. Once the position indicator is being generated, the controlunit operates the image sensor to observe the infrared spectral fieldforward of the image projecting device in step 85. After the imagesensor has been turned on, the position indicator is turned off in step86 and the control unit determines whether the infrared positionindicator is present on the image viewing field of the image sensor instep 88.

If the infrared position indicator is not within the image viewingfield, the control unit determines in step 90 that the device is notnear an ambient surface and calibration needs to be attempted in thefuture. Upon such determination, all correction Δ values are set todefault values in step 91. However, if the infrared position indicatoris viewable in the image field, the control unit determines the x and ycenter of the position indicator in the camera plane, as indicated instep 92.

Once the x and y locations of the position indicator are determined, thecontrol unit calculates an x-y indicator to camera correction Δ and anx-y camera to projector correction Δ in step 94, having

${C\text{-}P\mspace{14mu} {Parallax}\mspace{14mu} {Scale}\mspace{14mu} \left( {x,y} \right)} = \frac{{{Projector}\mspace{14mu} {center}\mspace{14mu} \left( {x,y} \right)} - {{Camera}\mspace{14mu} {center}\mspace{14mu} \left( {x,y} \right)}}{{{Camera}\mspace{14mu} {center}\mspace{14mu} \left( {x,y} \right)} - {{Emitter}\mspace{14mu} {center}\mspace{14mu} \left( {x,y} \right)}}$

using pre-determined x-y measurements made along the front end 14 of thedevice, where the Projector center(x,y) is the projector center axisposition, Camera center(x,y) is the camera center axis position, and theEmitter center(x,y) is the position indicating emitter center axisposition. In addition to the position of the position indicator in thecamera field, the control unit also determines in step 96 the size ofthe position indicator on the camera plane and computes the dimensioncorrection Δ in step 98. This process allows the control unit tocalibrate for both the position and size of the position indicator.

Finally, in step 100 the control unit determines the axis of rotation ofthe position indicator. As described previously, the position indicatoris preferably an anisotropic-shaped image, which allows the control unitto determine the rotation of the position indicator. The rotationalangle is computed and is stored as another correction Δ, as indicated instep 102. The calibration steps shown in FIG. 7 a allow the control unitto determine spatial correction Δ quantities that are used to adjust theresults of the spatial analysis conducted by the device.

As described previously, FIG. 4 a shows a first image 76 (virtual cat)projected by device 10 a, and a second image 78 (virtual dog) beingprojected by a second device 10 b. As one can see, the virtual dog isinteracting with the virtual cat in a playful manner. Device 10 a anddevice 10 b achieve this feat by transmitting spatial and characterattribute information as wireless messages back and forth to each other,while concurrently projecting the visible cat and dog respectively.

FIG. 4 b shows device 10 a shining the position indicating emitter atthe wall for a brief period, typically under 0.10 seconds. Preferably,the infrared beam from the emitter would be modulated, typically in therange of 1 kHz to 40 kHz, so as to provide a distinct signal orfingerprint of the light allowing a reduction of ambient noiseinterference. At the same time, the second device 10 b image sensor andcontrol unit filters out all infrared light that is not being modulatedusing signal analysis techniques. Many image sensors today can achievesuch high frames rates using Region of Interest read-outs. As a result,the modulated infrared light reflected from a wall would bedistinguished from the surrounding ambient infrared light, such as heatfrom a window, incandescent lights or a wall heat radiator. In addition,the wireless transceiver from device 10 a transmits a messageessentially reading:

  Device_Id=device 10a unique identifier Device_Position_Indicator = ONObject_Id=Fuzzy Object_Type=cat Object_Orientation=sitting and facingforward Image_Displacement=0, −2 Image_Dimension=10, 20Image_Rotation_Angle=0

The infrared beam from device 10 a is received by device 10 b. That is,device 10 b contains the CMOS camera that has an infrared camera view 64b of the wall on which the reflected infrared position indicator 60 aappears. The result is a hotspot or pixel region that is lit up on CMOScamera. Hence, the position of the device 10 a projected cat relative todevice 10 b projected dog may be determined. Device 10 b has its controlunit read the pixels of its CMOS camera and sees there is a brightly-litpixel region, defined by high value pixels (where pixel binary valuesare in proportion to brightness) on the CMOS. Subsequently, the device10 b has its microprocessor-based control unit determine the cat's x-ycoordinates by:

cat_relative_x = device10a_infrared_x - device10b_ camera_center_xcat_relative_y = device10a_infrared_y - device10b_ camera_center_y

-   where cat_relative_x is the cat's x-coordinate relative to the dog's    x-coordinate cat_relative_y is the cat's y-coordinate relative to    the dog's y-coordinate device10 a_infrared_x is device 10 a infrared    position indicator x-coordinate device10 a_infrared_y is device 10 a    infrared position indicator y-coordinate device10 b_camera_center is    device 10 b camera center x-coordinate (origin device10    b_camera_center is device 10 b camera center y-coordinate (origin)

In addition, the control unit is able to read the device 10 a wirelessmessage (defined above) by using its wireless transceiver. In theexample above, the control unit is able to interpret the messageindicating a cat is nearby.

Besides the determination of a relative x-y center coordinate, therelative size of the projected image from device 10 a can be determinedfrom the size of device 10 a's infrared position indicator. Of note, thesize of the observed infrared position indicator is directlyproportional to the size of the projected image. Whereby, if the size ofthe detected infrared position indicator from device 10 a is 10% largerthan the size of infrared position indicator from device 10 b, thendevice 10 a projected image is 10% larger than the device 10 b projectedimage.

In addition, the relative angle of rotation along the z-axis may bedetermined. Since each device projects an anisotropic shaped infraredposition indicator, the device can compute the relative rotation angleabout the z-axis.

FIG. 4 c shows device 10 b shining an infrared beam at the wall for abrief period, typically under 0.10 seconds. In addition, the wirelesstransceiver from device 10 b transmits a message essentially reading:

  Device_Id=device 10b unique identifier Device_Position_Indicator=ONObject_Id=Rover Object_Type=dog Object_Orientation=standing and facingleft Object_Activity_Device_Id=device 10a unique identifierObject_Activity= licking Image_Displacement=0, 0 Image_Dimensions=30, 20Image_Rotation_Angle=0

The infrared beam from device 10 b is received by device 10 a. Thedevice 10 a CMOS camera, which has an infrared camera view 64 a of thewall on which the reflected infrared position indicator 60 b appears.The result is a hotspot or pixel region that is lit up on CMOS camera.Hence, the position of the device 10 a projected dog, which is thecenter of the infrared position indicator 60 b, relative to device 10 aprojected cat may be determined. Device 10 a has itsmicroprocessor-based control unit read the pixels of its CMOS camera andsees there is a brightly-lit pixel region, defined by high value pixels(where pixel binary values are in proportion to brightness) on the CMOS.Subsequently, the device 10 a has its control unit determine the dog'sx-y coordinates by:

dog_relative_x = device10b_infrared_x - device10a_camera_center_xdog_relative_y = device10b_infrared_y - device10a_camera_center_y

-   where dog_relative_x is the dog's x-coordinate relative to the cat's    x-coordinate dog_relative_y is the dog's y-coordinate relative to    the cat's y-coordinate device10 b_infrared_x is device 10 b infrared    position indicator x-coordinate device10 b_infrared_y is device 10 b    infrared position indicator y-coordinate device10 a_camera_center_x    is device 10 a camera center x-coordinate (origin) device10    a_camera_center_y is device 10 a camera center y-coordinate (origin)

In addition, the control unit of device 10 a is able to read device 10 bwireless message (defined above) by using its wireless transceiver. Inthe example above, the control unit is able to interpret the messageindicating the dog is licking the cat.

The software on device 10 a determines that whenever a dog is nearby,the cat will attempt to interact. Once the device 10 a determines a dogis nearby, the control unit sends video content to its projector showingthe cat squeamishly squinting its eye as the dog licks its face.

Ten seconds later, device 10 a may have its control unit send videocontent to its image projector to show the cat pouncing on the dog. Inaddition, control unit will send a message via its wireless transceiverthat the cat is pouncing on the dog, and sends a command to soundsynthesizer to playback a loud cat meow. Device 10 b then reads thewireless message and sends video content to its projector showing thedog crouching down and hiding its face. The sound synthesizer may thenbe activated by the control unit to playback a whimpering dog sound.

Understandably, the exchange of communication between the devices, andsubsequent graphic and sound responses can go on indefinitely, or atleast until one of the device's batteries or rechargeable power supplyis depleted.

To prevent the infrared light probing technique described above fromcausing confusion when multiple devices are actively projecting theirrespective images onto the same wall, there are various techniques toensure effective communication, such as but not limited to:

1) All of the devices strobe their vicinity with infrared light in around robin fashion. That is, when one device is illuminating itsvicinity with infrared light, all other devices are only observing thelight and are not emitting infrared light.

2) Each device modulates the infrared light at a different rate (i.e. 1kHz, 3 kHz, etc.). Whereby, the infrared light from each device isdifferentiable.

Additional play value may be created with more character attributes,which may also be communicated to other characters. For example, acharacter attribute may be “strength,” a magnitude related to the healthof the dog, where the greater the strength value, the healthier thecharacter. Play interaction may affect a character's strength magnitude.As in the preceding message communications, if the dog is pounced on bythe cat, the dog's strength may be reduced. Whereby, the owner of thedog (the person holding the device) should take the dog to its waterdish to have a drink of fresh water, which increases the dog's strengthmagnitude. Understandably, a substantial number of character attributesand behaviors may be defined in the software for greater entertainmentvalue.

In the first embodiment of the invention shown in FIGS. 4 a-4 c, theimage projecting device interacts with another device such that theimage from a first device interacts and responds to either the image orthe movement of an image from the other device. FIG. 7 b is a simplifiedflowchart showing the interaction that occurs during use of a pair ofthe image projecting devices 10.

Initially, the wireless transceiver of the first device receives amessage from the second device that indicates that the positionindicating emitter of the second device is turned on, the imagedisplacement from the second device, the image dimensions from thesecond device and the image rotation angle from the second device, asshown in step 114. The image description received by the first devicehas already been correction Δ adjusted by the second device. Thus, theimage description is relative to the position indicator and exactly mapsto the projected image of the second device. Once the first devicereceives this information, the first device turns on its image sensor,which may be a CMOS camera, and observes the infrared spectral view infront of the first device, as shown in step 116. If the device sensesthe infrared position indicator from a second device in step 118, thefirst device determines the x and y center of the position indicator onthe sensor plane in step 120. In steps 121 and 122, the first devicedetermines the x and y dimensions and angle of rotation of the positionindicator on the sensor plane.

Based upon the determined parameters, the first device creates arectangle composed of four points, centered on the Cartesian originO(0,0), shown in step 123. The rectangle corresponds to the projectedimage shape of the second device. Subsequently, the first devicetransforms the projected image shape points onto its own image plane instep 124. The coordinate transformation M is completed using standard 2Dcoordinate transformation, where the homogenized coordinate 3×3 matricesare defined:

$\begin{matrix}{{T\left( {t_{x},t_{y}} \right)} = {\begin{bmatrix}x^{\prime} \\y^{\prime} \\1\end{bmatrix} = {\begin{bmatrix}1 & 0 & t_{x} \\0 & 1 & t_{y} \\0 & 0 & 1\end{bmatrix} \cdot \begin{bmatrix}x \\y \\1\end{bmatrix}}}} & {Translation}\end{matrix}$ $\begin{matrix}{{R(\theta)} = {\begin{bmatrix}x^{\prime} \\y^{\prime} \\1\end{bmatrix} = {\begin{bmatrix}{\cos (\theta)} & {- {\sin (\theta)}} & 0 \\{\sin (\theta)} & {\cos (\theta)} & 0 \\0 & 0 & 1\end{bmatrix} \cdot \begin{bmatrix}x \\y \\1\end{bmatrix}}}} & {{Rotation}\mspace{14mu} {about}\mspace{14mu} {origin}\mspace{14mu} {O\left( {0,0} \right)}}\end{matrix}$ $\begin{matrix}{{S\left( {s_{x},s_{y}} \right)} = {\begin{bmatrix}s_{x} & 0 & 0 \\0 & s_{y} & 0 \\0 & 0 & 1\end{bmatrix} \cdot \begin{bmatrix}x \\y \\1\end{bmatrix}}} & {Scaling}\end{matrix}$

A rectangle of four points results, describing the boundary or outlineof the projected image from the second device contained on the imageplane of the first device, although clearly other methods arecontemplated as being within the scope of the invention.

Once these values have been determined, the first device determines instep 126 whether the projected image from the second device has collidedwith the image from the first device, using a standard collisiondetection algorithm known in the art. In the embodiment described in thepresent application, if the two images have not collided, the processends in step 128. However, if the two images collide, the first devicegenerates video and sound in step 130 and projects a selected video thatcorresponds to the collision of the two images. The same processdescribed above for the first device is also carried out with respect tothe second device during the interaction of the pair of images.

If the system determines in step 118 that there is not a positionindicator within the sensor plane, device 1 determines that theprojected image from device 2 is not in the vicinity, as shown in step134.

Further interaction capabilities can be appreciated in knowing that thedevice's projected image can interact with real world objects. In FIG. 5a, a real-world person 104 is reaching out to the projected virtualimage 106 of a dog from device 10. As shown in FIG. 5 b, device 10 canobserve the person's hand 108 using the CMOS camera infrared imageviewing field 64. In such an application, the control unit of the device10 computes the person's hand relative x-y coordinate and responds bysending video content to the projector. As a result, the virtual dogturns around, wags its tail, and licks the real-world person's hand, asseen in FIG. 5 c.

Further interaction capabilities can be appreciated in knowing that thedevice 10 can optionally communicate with an external device 110, whichis connected to the internet network 112, shown in FIG. 16. In such anexample, the control unit of the device is able to send and retrievedata from its wireless transceiver, which transmits and receives datafrom the external device 90.

As pointed out earlier, the image projecting device is a standalonedevice, requiring no external hardware resources. However, one can takeadvantage of additional hardware and data resources provided by anexternal device 110. Such benefits may include access to: 1) greaterdata storage: 2) faster and more capable data processing capability; 3)more varied digital image, video, and sound content; 4) local intranetand worldwide internet data storage and processing resources; and 5) anadditional graphic display, perhaps having greater display area andresolution.

Further, if the external device 110 is a video game machine connected toa large video display, the projected image from the device 10 can bepointed at the external device's video display. Subsequently, thedevice's projected image could interact with the video display's image.For example, if the device's projected image was a dog, and the videodisplay subject matter was a garden of flowers, a player may sweep thehand-held projector device in a horizontal motion, and see an animateddog go rollicking through the garden of flowers.

Understandably, the projected image's subject matter, animation and playexperience may vary in myriad ways. The device is capable of projectingan image of characters, objects, props, background scenery or anythingthat may be visually represented. As an example, images of a dog, cat,goldfish, race cars, spaceships, dolls, and ninja fighters should all beconsidered among the possibilities. In addition, a character, such as adog, may appear with other animated characters, such as a flutteringbutterfly, or props such as fire hydrant. Finally, the device mayproject helpful graphic tools such as cursors, dialog balloons, popuppanels, and icons to facilitate complex play interactivity.

Further interaction capabilities can be appreciated in knowing that thedevice can display a collection of icons and menus, enabling quick andeasy navigation of play selections. For example, dialog panels mayappear superimposed on projected image, further facilitating one inselection of various options at each step of play interaction.

Further interaction capabilities can be appreciated in knowing that theprojected image can zoom in and out as a player moves the device towardsor away from the wall. For example, consider a player holding the device10 and standing in front of a wall, and the projected subject matter isa realistically rendered, full-size virtual door. If the player walkstowards the wall, the projected image of the virtual door willunfortunately shrink in size and destroy the visual realism. To avoidthis, the device updates the projected image periodically (e.g., every1/30 second) with the appropriate image zoom relative to the projectiondistance. Thus, as the player walks towards the virtual door, the visualscale of the door remains constant, giving the impression that theplayer is moving towards a full-size door. To carry out this function,control unit 24 reads the range locator 51, determines the distance tothe wall, and computes the necessary logical image zoom required tomaintain the scale of the projected image.

In addition to the embodiment described that allows two images from twoimage projecting devices to interact with each other, it is alsocontemplated that the image projecting device could be utilized aloneand used to project an image 136 that creates an environment around theuser 104, as shown in FIG. 8 a. In the example shown in FIG. 8 a, theimage 136 includes a castle door 138 having a doorknob 140. FIG. 8 aillustrates the person 104 interacting with the image 136 by making anon-contact hand gesture within the light beam 142 of the projector,causing a shadow cursor 144 to appear.

In the example shown in FIG. 8 a, if the player 104 wishes to open thedoor 138, the user 104 would place a hand 108 in front of the projectorto interrupt the projector's beam of light and create a shadow. Thisshadow acts as a visual cursor 144 so that the user can position theirfinger in the approximate spot in front of the projection device. Sincethe image sensor of the device 10 senses the infrared spectral view infront of the device 10, the image sensor can determine that a finger isbeing moved at a specific location, as shown in FIG. 8 c. The controlunit turns on the infrared illuminating emitter 74, which improves thevisual contrast between the finger and the wall. The control unit thenreads a sequence of captured image frames from the image sensor, perhapsten image frames at about ¼ second intervals. Whereupon, the controlunit compares the individual image frames and converts the moving “litup” region of pixels into an x-y coordinate on the projected image shownin FIG. 8 b. Based upon the location of the user's finger, the softwareof the control unit converts the x-y coordinates of the players handinto action. For example, if the control unit determines that the x-ycoordinate of the player's hand corresponds to the doorknob of the door,the control unit sends video content to the projector causing thevirtual door to open, as illustrated in FIG. 8 d.

In addition to the embodiment shown in FIG. 8 a, various other types ofhand gestures and images are contemplated as being within the scope ofthe present disclosure. In the environment shown in FIG. 11, the controlunit generates a virtual saber 146 that can be moved by the player 104by moving the player's hand 108 in a chopping motion 148. The result isan animated image of the virtual saber 146 being thrust toward anadvancing virtual dragon 136.

As described previously with reference to FIG. 10, the image projectionfield 48 has a projection angle that is substantially less than theimage viewing field 64. In this manner, the image sensor is able todetect movement of a player over a wider field of view than theprojected image 48. As seen in FIG. 8 c, the image viewing field 64 issubstantially greater than the image projection field 48 such that theplayer's hand 108 can be detected well before the player's hand reachesthe image projection field 48. In the preferred embodiment, the optimumCMOS camera viewing range should be at least twice that of the projectedlight cone. Typically, the projected light cone 48 would have an angleof about 20°-60° and the CMOS camera view would have an angle of about90°-120°.

In addition to the non-contact hand gestures contemplated in FIGS. 8 b-8d, it is contemplated that the image projecting device could also beutilized to detect when a person contacts the wall or surface that theimage is being projected onto. The preferred approach of carrying outsuch function is to use a source of infrared light emitted by thedevice, such as the illuminating emitter 74 shown in FIG. 1, to producea strong shadow effect. Based upon the shadow effect, the image sensorof the device can determine when the player contacts the image viewingsurface and, upon detecting contact, carry out the required process.

FIGS. 6 a and 6 b illustrate a person 104 standing in front of a viewingscreen 150 onto which an image may be projected. The screen 150 could bea wall, movie screen or any other type of surface onto which an imagecan be projected. FIG. 6 c illustrates what the image sensor, such asthe infrared CMOS camera, observes in the infrared spectrum as a user isabout to make a contact hand gesture. As shown, the human hand can bedetected using a standard CMOS camera, which is sensitive to infraredlight up to 1 micron wavelength. For purposes of teaching, if the sensorhad sensitivity to infrared light beyond 1 micron, the image contrastwould increase. That is, since the human hand 108 is warmer than itssurroundings, the hand 108 would appear “lit-up” and glowing against adark background. Although the image in FIG. 6 c indicates that the CMOScamera is able to detect the presence of a human hand, the image of FIG.6 c alone cannot determine whether the hand is actually in contact withthe screen 150.

To detect the contact between the player's hand 108 and the screen 150,the control unit of the image projecting device 10 can turn on theilluminating emitter 74 (FIG. 1) to cause the player's hand 108 to castan infrared shadow 152, as shown in FIG. 6 d. Since the shadow 152 iscreated by the infrared illuminating emitter 74, the shadow is notvisible to the user.

FIGS. 6 e-6 h illustrate the movement of the player's hand 108 intocontact with the display screen 150. As can best be illustrated in FIG.6 h. as the player moves his hand 108 into contact with the screen 150,the shadow 152 trails off to a sharp point at the end of the finger 154.When the image sensor, such as the CMOS camera, detects the wholeocclusion of the shadow by the player's hand or finger tips, the controlunit determines that the player has touched the display screen 150 andresponds accordingly. As an example, the player 104 may be depressing abutton, area or any other section of the image displayed by the deviceas is desired. When the user moves his or her finger into position todepress the select area, the control unit of the image projecting device10 changes the image being projected in direct response to the player'scontact.

The image projecting device of the present invention can perceive amultitude of non-contact and contact hand gestures from the user'ssingle or two free arms. These gestures may include the simulation ofriding a virtual motorcycle, boxing a virtual opponent, climbing avirtual rope, scaling a virtual wall, swimming in virtual water, liftinga virtual barrel and/or hugging a virtual object. Further, the imageprojecting device can perceive two or more players making non-contacthand gestures near a single projector. This ability to detect themovement of more than one hand allows a broader range of interactiveplay to be possible.

FIG. 9 a shows a useful high-level flow diagram of the steps the controlunit makes when observing, detecting, analyzing, and responding to ahand gesture. Initially, the Video Frame Grabber module reads the CMOScamera frame and stores an image frame for future processing in step300. The Foreground and Background Region Segmentation module scans andsegments the image frame into foreground and background regions, lookingfor distinct qualities such as variation in brightness, shown in step302. Once the image frame has been scanned and segmented, the HandDetection and Tracking module seeks out the temporal and spatial pointsof interest, such as a blob region moving quickly left to right,recording the position and velocity of such movements in step 304. TheGesture Analysis module then takes the recorded movements and tries tofind a match against a database of predefined gestures, using a HiddenMarkov Model, neural network, finite state machine, or various otherapproaches known in the art. If a match occurs, the hand movement isclassified as a specific type of gesture in step 306. If a gesture hasbeen determined in step 308, then a complimentary video and soundresponse is generated in step 312 and produced in step 314.

FIG. 9 b illustrates the operational flow steps used by the control unitof each of the image projecting devices when the image projecting deviceis used to detect and respond to either a non-contact hand gesture or acontact hand gesture. Initially, the image sensor, which may be a CMOScamera, observes the infrared spectral view in front of the device, asshown in step 170. The observed image from the CMOS camera is storedinto memory in step 172 and the wide angle infrared illuminating emitteris turned on in step 174. As described in FIG. 6 d, the infraredilluminating emitter creates an infrared shadow 152 that the CMOS cameraobserves in step 176. The image including the infrared shadow is storedinto image frame 2, as shown in step 178. After the image has beenrecorded, the infrared illuminating emitter is turned off in step 180.

Once the first and second image frames have been stored, the controlunit performs a visual image subtraction in step 182. For each imageframe cell in the x-y coordinate system, the control unit determineswhether the image frame cell intensity significantly differs from thebackground intensity in step 184. The background intensity could becomputed by taking the mean average of local cell intensity, or othertechniques known in the art. If the cell intensity differs from thebackground intensity, a human hand has been detected and the cell isclassified as “human”. However, if the cell intensity does not differfrom the background intensity, the system determines in step 188 whetherthe cell intensity is approximately zero. A representation ofapproximately zero indicates that the cell is a shadow and is labeled assuch in step 190.

If the cell is neither human nor a shadow, the system determines whetherany additional cells remain and the process is continued for eachindividual cell. In step 194, the image frame 4 description of the humanhand and shadow regions are analyzed to determine in step 196 whether ahuman hand has been detected. If a human hand has not been detected,then the system determines in step 198 that no contact or non-contactgesture has been made. However, if a human hand has been detected, thesystem determines in step 200 whether the shadow tapers off at the endpoint of the hand gesture. If the shadow tapers off, the systemdetermines in step 202 that a contact gesture has been made. However, ifthe shadow does not taper off, no contact gesture has been made, asillustrated in step 204. By utilizing the method described in FIG. 9 b,the system is able to determine whether or not a contact or non-contactgesture has been made by a player/operator.

FIG. 9 c illustrates the operational flow steps used by the control unitafter a non-contact hand gesture has been detected and classified by theGesture Analysis module. For reference, FIG. 11 shows the actual“chopping” like gesture being made by the user. Referring back to FIG. 9c, when the “chopping” gesture has been detected in step 330, thesegmented image frame (containing hand and background regions) isretrieved from the gesture engine, and copied into the image frame 1,shown in step 334. Subsequently, the segmented image frame 1 is searchedaround its perimeter, and a mid point PS is located in the human regionblob at the edge of the frame. Point PS represents the position of thebase of the human arm, shown in step 336. Using a standard regionscanning algorithm known in the art, the most distant point PE withinthe human region from point PS is located, as shown in step 338. PointPE represents the position of the hand's fingertips. A vector angle α iscomputed from Point PS to PE in step 340.

A graphic response can now be generated, as shown in step 346 of FIG. 9c. Initially, the “castle” background and a “dragon” object imagery isgenerated, transformed, and copied to image frame 2. The imagegeneration process very much relies on the orientation of the device tothe projected surface, as indicated in step 342. The previouslydetermined points PS and PE are then mapped onto image frame 2, as x-ycoordinates for the saber image and virtual arm image, shown in step344. The saber image and virtual arm image are rendered in image frame 2in step 352. A saber whooshing sound is retrieved from the sounddatabase in step 354, and sent the sound synthesizer for playback instep 356. Finally, the fully rendered image frame 2 is copied into theprojector video frame in step 358, for immediate light projection to theuser's delight.

Further interaction capabilities can be appreciated in an embodimentwhere there is means for the player to interact with the projected imageusing sound. The image projecting device shown and described includes amicrophone 49, as presented in FIG. 1 and FIG. 2, that is capable offurther enhancing the interactive play experience. As an example, theimage projecting device could project an image of a virtual door on awall. If the user reaches out and knocks on the virtual door, themicrophone of the image projecting device can respond to the knock byopening the projected virtual door.

Referring now to FIG. 12, further interaction capabilities can beappreciated by providing a supplementary passive gesture glove 156. Inthe embodiment shown in FIG. 12, the gesture glove 156 is made of amaterial that is retro-reflective to the infrared spectrum. Since thedevice 10 includes an infrared light emitter, the device 10 can sensethe reflective infrared light coming from the glove 156. As a result, aplayer can move the glove 156 in 3D space and the device 10 willinterpret the position, velocity and orientation of the glove 156.

Understandably, the passive gesture glove 156 of FIG. 12 is not the onlypossible embodiment. The glove 156 represents only one example of ameans for a person to interact with the projected image. The passivegesture component could also be a ring, bracelet, shoe or hat worn bythe user.

FIG. 13 illustrates another type of gesture device 158. In theembodiment shown in FIGS. 13 and 14, the gesture device 158 is wornaround the player's wrist and can communicate through wirelesscommunication to the image projecting device 10. As shown in FIG. 14,the gesture device 158 can include a three axis accelerometer 159, awireless transceiver 160, a microprocessor 162 and a battery powersupply 164. When the player is using the supplementary gesture device158 positioned on their right hand (FIG. 13), the player can make aglancing blow at the image by quickly raising the right hand. Such amovement will cause the gesture device's microprocessor to read theaccelerometer 159 and send a message along with the accelerometer'svalues to the wireless transceiver 160. This information is received bythe image projecting device 10 through the wireless transmission 166shown in FIG. 13. In this manner, further movements of the player'shands can be more quickly and easily detected by the image projectingdevice 10.

As described previously, images from more than one image projectingdevice can respond to each other upon one of the devices detecting theimage of another device. In addition, it is contemplated that the imagesfrom more than one image processing device can be synchronized toprovide a single, coherent image result. This single, coherent imageresult is important in game play since there may be multiple players,each holding a device 10, interacting in the same 3D virtual space. Thatis, when two projected images happen to overlap, the user should not seetwo unrelated images, but rather a single, coherent image of a commonvirtual space.

FIG. 15 a illustrates a projected visual image 168 of a castle from adevice 10 a. Nearby, a second device 10 b is also projecting a visibleimage of a different portion of the same castle. As the images movecloser together, both devices need to synchronize their images and cancarry out such function as follows. As shown in FIG. 15 b, device 10 ais shining its infrared position indicating emitter to create theposition indicator 60 a on the wall. The position indicator 60 a appearswithin the infrared camera viewing field 64 b of the second device 10 b.The first infrared emitter 60 a not only enables a relative x-ycoordinate to be determined, but also the relative angle and relativedimension of one image with respect to another. Upon detecting theposition indicator 60 a, device 10 b computes the relative x-ycoordinates, relative angle, and relative dimension of the projectedimage of device 10 a. As a result, device 10 b can adjust its image bydigitally panning, rotating, zooming in or out, and clipping until imagealignment is achieved. FIG. 15 c shows the projected image 169 of device10 b being clipped along the edge for overlap with the image 171 fromdevice 10 a. FIG. 15 d illustrates the resulting combined image 173 fromboth devices 10 a and 10 b.

In addition to the interaction between the images from each device, itis also contemplated that the devices held by multiple players caninteract with each other in the same virtual space. As an example,assume a first player has killed a dragon at a first virtual location.If the second player travels to the same location, the second playerwill not see a live dragon, but rather will find an already slaindragon. As a result, players may work together to solve common tasks.

Such a feat is carried out using data synchronization by way of amaster-slave device protocol. Specifically, one of the devices isdefined as the master device, which holds the up to date “master” copyof the state of all local virtual objects, such as object type,position, velocity, strength, etc. When a slave device alters a virtualobject, its wireless transceiver sends a message to the master deviceindicating the new data state of the object. The master device thenupdates the corresponding object in its memory space and wirelesslybroadcasts the data state of the virtual space and objects to all of thesurrounding slave devices, using its built-in wireless transceiver.Typically, the master device broadcast rate would be every 1/10 to1/1,000 of a second so that the nearby slave devices are kept up to datewith timely data.

FIGS. 17-19 illustrate an alternate embodiment of an image projectingdevice, as referred to by reference numeral 206. In the secondembodiment shown in FIGS. 17-19, similar reference numerals are utilizedfor common components with respect to the first embodiment shown inFIGS. 1-3.

As can best be seen in FIG. 18, the projector 42 is a low-cost LED basedprojector system. Specifically, the projector 42 includes a white LED208 that shines white visible light on a Fresnel lens 210, whichcollimates the light into parallel rays. The light rays are thentransmitted through a monochrome LCD 212 that allows light to pass onlywhere the image pixels are transparent and the light is blocked wherethe image pixels are opaque. The modulated light then passes through amulti-color plastic film sheet 214, which is an inexpensive means tocolor the projected image, and onto the front Fresnel lens 216, whichcauses light to converge toward the lens apparatus. The light is thentransmitted through an optical focusing lens 218 mounted within an outerhousing 220 that is threadedly received along the neck 222. The rotationof the housing 220 allows the image to be focused at the desired focallength depending upon the distance of the image projecting device 206from the image surface. The use of the LED 208 as the projector is alow-cost version of the image projecting device 206, as compared to thelaser-based projector of the first embodiment.

In addition to the replacement of the projector 42, the embodiment shownin FIG. 18 includes an alternate version of an image sensor 62. As bestseen in FIG. 17, the image sensor 62 has been replaced by six, low-cost,infrared sensors 224. As illustrated in FIG. 18, each of the sensors 224sits deep in a round-shaped focusing waveguide 226 that creates aconstant viewing angle for each of the sensors 224. Preferably, theround-shaped focusing waveguide 226 is a cavity or depression in thebody of the device 206. Typically, each of the sensors 224 has a viewingangle of anywhere between 10°-30°, which is dictated by the waveguide226.

Each of the sensors 224 is in operative communication with the controlunit 24 such that information from the sensors can be received andinterpreted by the control unit 24. Like the first embodiment, thesecond embodiment of the image projecting device 206 also includes aposition indicating emitter 52 positioned behind a waveguide 56. Inaddition, as shown in FIG. 17, the second embodiment includes a secondposition indicating emitter 53 behind a waveguide located below positionindicating emitter 52.

In the embodiment shown in FIG. 18, the longitudinal axis of eachround-shaped focusing waveguide 226 is not parallel, but each axisexists on a diverging angle from the position indicating emitter 52longitudinal axis. This results in the sensor's viewing angles beingdivergent, allowing the device to observe an overall larger field ofview with greater spatial positional resolving capabilities. Arepresentation of the viewing fields 234 for the sensors 224 is shown inFIG. 21 a.

In the embodiment shown in FIG. 17, the device includes a keypad 228that includes a series of touch-sensitive buttons for providing variousdistinct operating functions.

FIG. 20 shows a virtual spaceship projected by device 206 a, and avirtual rocket being projected by device 206 b. Assume that the user ofdevice 206 a has pressed touch-sensitive keypad 228 button to “firelaser.” The result is the microprocessor of device 206 a sends videodata to its projector system showing its projected image of a spaceshipfiring a laser, as shown in FIG. 20. Immediately following the visiblelight projection of the spaceship firing a laser, FIG. 21 a shows device206 a shining a first infrared beam at the wall for a brief period, suchas 1/100 second. This first beam originates from position indicatingemitter 52. In addition, the modulated infrared beam is encoded with amessage from device 206 a essentially reading:

  Device_Id= device 206a unique identifier Device_indicator_Position=ONObject_Id=Blaster Object_Type= spaceship Object_Orientation= Facingright Object_Activity= fired laser, 45 degree angle, r=20 unitsImage_Displacement=0, −2 Image_Dimension=22, 22 Image_Rotation_Angle=120

Once 1/100 second has elapsed, device 206 a turns off the first beam andturns on a second infrared beam for a 1/100 second duration. This secondbeam originates from position indicating emitter 53.

During which time, the first beam's infrared message from device 206 ais received by device 206 b. Specifically, device 206 b contains the sixinfrared sensors 224 that combine for the infrared sensor view 232 b ofthe wall on which the reflected infrared spot 230 a appears. The resultis that some of the infrared sensors will detect the reflected infraredlight, and the other sensors will not. Hence, the position of the device206 a projected spaceship, which is the center of the infrared spot 230,relative to device 206 b projected rocket may be computationallydetermined. Device 206 b has its control unit, via a standardanalog-to-digital converter, read the analog values of all six infraredsensors 224 and sees that several of the infrared sensors 234 havesignificantly higher binary values (where binary values are inproportion to brightness). Subsequently, the device 206 b has itsmicroprocessor determine the spaceship's x-y coordinates by:

spaceship_relative_x= device206a_infrared_x - device206b_camera_center_x spaceship_relative_y= device206a_infrared_y -device206b_camera_ center_y

-   where spaceship_relative_x is the spaceship's x-coordinate relative    to the rocket's x-coordinate spaceship_relative_y is the spaceship's    y-coordinate relative to the rocket's y-coordinate device206    a_infrared_x is device 206 a infrared beam's x-coordinate device206    a_infrared_y is device 206 a infrared beam's y-coordinate device206    b_camera_center_x is device 206 b camera center x-coordinate    (origin) device206 b_camera_center_y is device 206 b camera center    y-coordinate (origin)

In addition, the microprocessor of device 206 b is able to read device206 a encoded message (defined above) by using its six infrared sensors224. Since the infrared beam is encoded with a message, the infraredspot 230 a will fluctuate in magnitude in proportion to the magnitude ofthe encoded signal from device 206 a. Hence, the microprocessor controlunit will be able to convert its infrared sensor fluctuating brightnessinto the message indicating a spaceship has fired its laser.

About 1/100 second later, the second infrared beam of device 206 a isreceived by device 206 b. That is, the microprocessor of device 206 breads its six infrared sensors 224, and computes the x-y position ofinfrared spot 2311 a. Whereby, the microprocessor then computes therelative rotation angle of the spaceship by determining the vector angleof infrared spots 231 a and 230 a.

The software on device 206 b determines that given the angle and lengthof the laser from the spaceship, and the relative x-y position androtation angle of the spaceship, the rocket has been hit. Whereby device206 b has its microprocessor send video content to its projector showingthe rocket has been hit. In addition, the microprocessor of device 206 bincrements the rocket's damage attribute contained in its software toreflect the laser hit.

FIG. 21 b shows device 206 b shining an infrared beam 230 b at the wallfor a brief period, such as 1/100 second. In addition, the modulatedinfrared beam is encoded with a message from device 206 b essentiallyreading:

  Device_Id=device 206b unique identifier Device_Indicator_Position=ONObject_Id=Speedy Object_Type=rocket, Object_Orientation=Facing rightObject_Activity_Device_Id=device 206a unique identifierObject_Activity=award 10 points Image_Displacement=0, 0Image_Dimension=20, 20 Image_Rotation_Angle=45

The infrared message from device 206 b is received by device 206 a. Thatis, device 206 a contains six infrared sensors 224, which has aninfrared sensor view 232 a of the wall on which the reflected infraredspot 230 b appears. The result is sensors 234 a will detect thereflected infrared light, and the other sensors will not. Hence, theposition of device 206 b projected rocket, which is the center of theinfrared spot 230 b, relative to device 206 a projected spaceship may becomputationally determined. Device 206 a has its microprocessor, via astandard analog-to-digital converter, read the analog values of all sixinfrared sensors 224 and sees that infrared sensors 234 a havesignificantly higher binary values (where binary values are inproportion to brightness). Subsequently, the device 206 a has itsmicroprocessor determine the rocket's x-y coordinates by:

rocket_relative_x= device206b_infrared_x - device206a_camera_center_xrocket_relative_y= device206b_infrared_y - device206a_camera_center_y

-   where rocket_relative_x is the rocket's x-coordinate relative to the    spaceship's x-coordinate rocket_relative_y is the rocket's    y-coordinate relative to the spaceship's y-coordinate device206    b_infrared^(—)x is device 206 b infrared beam's x-coordinate    device206 b_infrared_y is device 206 b infrared beam's y-coordinate    device206 a_camera_center_x is device 206 a camera center    x-coordinate (origin) device206 a_camera_center_y is device 206 a    camera center y-coordinate (origin)

In addition, microprocessor of device 206 a is able to read device 206 bencoded message (defined above) by using its six infrared sensors 224.Since the infrared beam is encoded with a message, the infrared spot 230b will fluctuate in magnitude in proportion to the magnitude of theencoded signal from device 206 b. Hence, the microprocessor will be ableto convert its infrared sensor 224 fluctuating brightness back into themessage indicating the awarding of 10 points.

About 1/100 second later, the second infrared beam of device 206 b isreceived by device 206 a. That is, the microprocessor of device 206 areads its six infrared sensors 224, and computes the x-y position ofinfrared spot 231 b. Whereby, the microprocessor then computes therelative rotation angle of the rocket by determining the vector angle ofinfrared spots 231 b and 230 b.

The software on device 206 a then adds ten points to the user's score.In addition, device 206 a has its microprocessor send video content toits projector showing the 10 points have been added.

Understandably, the exchange of communication between the devices, andsubsequent graphic and sound responses can go on indefinitely or atleast until one of the device's batteries or rechargeable power supplyis depleted.

FIG. 22 illustrates another embodiment of the image projecting device 10as shown in FIG. 22. The embodiment shown in FIG. 22 is similar to theembodiment shown in FIG. 1 except that the embodiment shown in FIG. 22includes a color LCD 234 mounted to the top surface 18. Further, thedevice shown in FIG. 22 includes a toggle switch 236 mounted on thedevice body. During operation, the toggle switch 236 can be depressed toactivate the color LCD 234 while at the same time deactivating theprojector 42. The color LCD 234 facilitates viewing an image in abrightly lit area or when ambient viewing surfaces may not be available.

When a nearby wall or surface is available along with low lightconditions ideal for image projection, the user can depress the toggleswitch 236 to deactivate the color LCD 234 and activate the projector42.

FIG. 23 illustrates yet another alternate embodiment of the imageprojecting device 10. In the embodiment shown in FIG. 23, the deviceincludes a rotating display mode lever 238 on the outside of the devicethat the player may rotate in a clockwise or counterclockwise motionwithin a 45° range of movement. A pivoting light reflector 240 exists inthe path of the projected beam 242 from the projector 42 that allows thebeam to be deflected in various directions within the outer housing 12.The light reflector 240 is preferably constructed from a plastic ormetal material that is coated with a mirror-like reflective surfaceusing vacuum metal deposition or scintillation processes. The pivotinglight reflector may have a shaped or curved design, resulting in areflected beam of the desired divergent angle. The outer housingincludes a translucent display 244 that is capable of transmittingvisible light from its back to its front surface, enabling one to view adisplayed image. The frosted or diffused transparent material 244 may bemade of such things as injection-molded acrylic, polycarbonate oracetate.

During operation, the user may toggle between viewing the on-boarddisplay and viewing the projected display. For example, to view theon-board display 244, the user rotates the lever 238 counter-clockwiseuntil it reaches a stop position. As a result, the light reflector 240will reflect light from the projector 42 into the back of the display244 where it can be viewed by a user. If the user no longer wishes toview the display 244, the lever 238 is rotated in the opposite directionto move the light reflector 240 out of the projected beam of light 242.When the light reflector is rotated out of the beam 242, the light willpass through the transparent protective cover 246 and onto a nearbyviewing surface.

Referring now to FIGS. 24 and 25, thereshown is yet another alternateembodiment of the image projecting device, as referred to by referencenumeral 248. In the embodiment shown in FIGS. 24 and 25, the device 248has the general configuration and shape of a flashlight that includes ahandle 250 sized to receive a pair of batteries 252. The batteries 252are in contact with a contact 254 coupled to a touch pad 256. In theembodiment shown in FIG. 4, the device 248 includes four infrared photosensors 258 placed at four equally spaced points around the generallycircular outer circumference of the device. Each of the photo sensors258 is aligned with a waveguide 260, as shown by the section view ofFIG. 25. The waveguide 260 controls the viewing angle of each of theinfrared sensors 258.

In the embodiment shown in FIG. 25, the laser-projector of the firstembodiment is replaced by a collection of six white LEDs 262, atranslucent multi-image film 264 and a focusing multi-lens barrel 266.Typically, each of the LEDs 262 has a projection angle of about 5°-20°.The translucent multi-image film 264 is a collection of six distinctimages, lithographically printed with translucent ink on a transparentpolymer, such as acetate or acrylic. It is assumed that there is adistinct image forward of each white LED 262. Preferably, the lensbarrel 266 is of an injection-molded transparent and flexible polymerpart, made of acrylic, PET or polycarbonate. The lens barrel has beensegmented into six sectors, where each lens 268 is surrounded by anin-molded living hinge 270 allowing all of the lenses to expand andcollapse radially around the barrel center.

In the embodiment shown, the device includes a position indicatingemitter 52 that projects an infrared image through the center of thefilm 264 and onto the projection surface.

During operation of the device shown in FIGS. 24 and 25, the userinitially turns on the device by touching the touch-sensitive keypad256. Once turned on, the control unit 47 coordinates the activation anddeactivation of the LEDs 262. When the white LED 262 is activated, whitevisible light is projected forward, transmitted through the coloredimage film 264 that produces filtered color light that is refracted bythe optical lens 268, producing a viewable image on an ambient surface.A projected animated image is achieved by the projector system bydeactivating the light LED associated with the undesired image andactivating the white LED associated with the desired image. Bycoordinating the activation of the LEDs 262, the control unit 47 cancreate an animated image.

Focusing of the image on an ambient surface can be accomplished byturning the lens barrel 266 clockwise or counterclockwise until theimage becomes clear. When the user turns the lens barrel 266 clockwiseand toward the LEDs 262, the barrel cap 271 has a slight concavecurvature, enabling the projector light to pass through the opticallens, gradually converging as a single coherent image on a distant wall.When the user turns the lens barrel 266 in a counterclockwise directionaway from the LEDs 262, the barrel cap 271 has a more severe concavecurvature, enabling the projected light to pass through all of theoptical lenses, sharply converging as a single coherent image on anearby wall.

Various alternatives and embodiments are contemplated as being withinthe scope of the following claims particularly pointing out anddistinctly claiming the subject matter regarded as the invention.

1-25. (canceled)
 26. A first image projecting device of handheld size,comprising: a control unit; a projector operable to project a firstimage; a position indicating emitter operable to project a firstposition indicator on a display surface; a wireless transceiver operableto transmit a first message; and an image sensor operable to view thedisplay surface and detect a second position indicator from a secondimage projecting device, wherein the control unit modifies the firstimage projected by the projector based upon the detected second positionindicator from the second image projecting device.
 27. The first imageprojecting device of claim 26, wherein the control unit determines arotation angle of the detected second position indicator from the secondimage projecting device, and wherein the control unit modifies the firstimage projected by the projector based upon the rotation angle of thesecond position indicator from the second image projecting device. 28.The first image projecting device of claim 26 further comprising aspeaker and a sound synthesizer controlled by the control unit, whereinaudible sound is generated based upon the detected second positionindicator from the second image projecting device.
 29. The first imageprojecting device of claim 26, wherein the first message comprises adevice ID having a unique identifier.
 30. The first image projectingdevice of claim 26, wherein the first message comprises image dimensionsof the first image.
 31. The first image projecting device of claim 26,wherein the message comprises an object type.
 32. The first imageprojecting device of claim 26, wherein the message comprises an objectactivity.
 33. The first image projecting device of claim 26, wherein thefirst message comprises a description of an object presented in thefirst image.
 34. The first image projecting device of claim 26, whereinthe first message comprises a description of a character presented inthe first image.
 35. The first image projecting device of claim 26,wherein the control unit determines that a second image projected by thesecond image projecting device is in the vicinity of the first imageprojected by the first image projecting device, and wherein the controlunit modifies the first image based upon the second image being in thevicinity of the first image.
 36. The first image projecting device ofclaim 26, wherein the control unit determines that a second imageprojected by the second image projecting device has collided with thefirst image projected by the first image projecting device, and whereinthe control unit modifies the first image based upon the collision ofthe first image and the second image.
 37. The first image projectingdevice of claim 26, wherein the wireless transceiver receives a secondmessage from the second image projecting device, and wherein the controlunit modifies the first image based upon the received second message.38. The first image projecting device of claim 37 wherein the secondmessage comprises a description of an object presented in the secondimage.
 39. The first image projecting device of claim 37, wherein thesecond message comprises a description of a character presented in thesecond image.
 40. A method of integrating the operation of a first imageprojecting device and a second image projecting device, the methodcomprising: generating a first image from a first projector of the firstimage projecting device; generating a position indicator from a positionindicating emitter of the first image projecting device; transmitting amessage from a first wireless transceiver of the first image projectingdevice; operating an image sensor of the second image projecting deviceto detect the position indicator; and modifying a second image from asecond projector of the second image projecting device based upon thedetected position indicator.
 41. The method of claim 40 furthercomprising: determining the position of the position indicator by thesecond image projecting device; determining the location of the firstimage by the second image projecting device based upon the position ofthe position indicator; and modifying the second image from the secondprojector of the second image projecting device based upon thedetermined location of the first image.
 42. The method of claim 40further comprising: determining a collision between the first image andthe second image by the second image projecting device; and modifyingthe second image from the second projector of the second imageprojecting device based upon the determined collision between the firstimage and the second image.
 43. The method of claim 40 furthercomprising: receiving the message by a second wireless transceiver ofthe second image projecting device; modifying the second image from thesecond projector of the second image projecting device based upon themessage received.
 44. The method of claim 43, wherein the messagecomprises a device ID having a unique identifier.
 45. The method ofclaim 43, wherein the message comprises image dimensions of the firstimage.
 46. The method of claim 43, wherein the message comprises adescription of an object presented in the first image.
 47. The method ofclaim 46 further comprising: detecting the description of the objectpresented in the first image by the second image projecting device; andmodifying the second image from the second projector of the second imageprojecting device based upon the description of the object presented inthe first image.
 48. The method of claim 43, wherein the messagecomprises an object type.
 49. The method of claim 43, wherein themessage comprises a description of a character presented in the firstimage.
 50. The method of claim 49 further comprising: detecting thedescription of the character presented in the first image by the secondimage projecting device; and modifying the second image from the secondprojector of the second image projecting device based upon thedescription of the character presented in the first image.
 51. A firstimage projecting device of handheld size, comprising: a control unit; aprojector operable to project a first image controlled by the controlunit; a position indicating emitter operable to project a first positionindicator on a display surface; an image sensor operable to view thedisplay surface and detect a second position indicator from a secondimage projecting device; and a wireless transceiver operable to receivea message from the second image projecting device, wherein the controlunit modifies the first image projected by the projector based upon thedetected second position indicator and the message received from thesecond image projecting device.
 52. The first image projecting device ofclaim 51, wherein the message comprises a device ID having a uniqueidentifier.
 53. The first image projecting device of claim 51, whereinthe message comprises image dimensions of the first image.
 54. The firstimage projecting device of claim 51, wherein the message comprises animage rotation angle.
 55. The first image projecting device of claim 51,wherein the message comprises an object type.
 56. The first imageprojecting device of claim 51, wherein the message comprises an objectactivity.
 57. The first image projecting device of claim 51, wherein themessage comprises character attribute information.
 58. The first imageprojecting device of claim 51, wherein the control unit determines theposition of the second position indicator from the second imageprojecting device, and wherein the control unit modifies the first imageprojected by the projector based upon the position of the secondposition indicator and the message received from the second imageprojecting device.
 59. The first image projecting device of claim 51,wherein the control unit determines the rotation angle of the secondposition indicator from the second image projecting device, and whereinthe control unit modifies the first image projected by the projectorbased upon the rotation angle of the second position indicator and themessage received from the second image projecting device.
 60. The firstimage projecting device of claim 51 further comprising a speaker andsound synthesizer controlled by the control unit, wherein audible soundis generated based upon the detected second position indicator and themessage received from the second image projecting device.